Skip to content

ci: prewarm the assertion-rewrite cache before backend shards - #39398

Draft
Zedmor wants to merge 1 commit into
mainfrom
ci/rewrite-only-prewarm
Draft

Zedmor wants to merge 1 commit into
mainfrom
ci/rewrite-only-prewarm

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A full backend CI run spends less time collecting tests before the first test starts.

Every xdist worker on a backend shard collects the whole shard and repeats pytest's assertion rewrite for every test file, because CI starts with an empty bytecode cache. A Linux worker collects for a median 252 s and a Windows worker for 108 s (9 green full-suite runs on main, 2026-11-18). #39341 measured a serial pytest --collect-only prewarm and dropped it: it cost 104-115 s and saved 42-75 s, because it imports every test module in one process.

Why it matters

Collection is paid by every worker of all 16 shards on every full run, before any test runs. It is CPU-bound, so it does not shrink when workers are added.

Not a goal

What changed (motivation → approach → change)

Workers repeat the same rewrite → do it once, in parallel, without importing anything → a new step before Run tests in backend-test and backend-test-windows.

This is an experiment with a ship gate. From #39368: "Ship only if (prewarm step time + new collection time) is lower than the old collection time on the median Linux shard by at least 30 s." The PR stays a draft until its own full-suite run is measured. Placeholder: the per-shard table (prewarm step time and collection_end.elapsed, 8 Linux + 8 Windows shards, against the median of the last 3 green full-suite runs on main) will be added as a PR comment after CI. If the gate fails, this PR is closed with the numbers.

  • scripts/ci_rewrite_prewarm.py (new). Takes --file-shards, --file-shard and --file-shard-pins, as the test step does. Splits the files over one spawn process per core, largest first.
  • The step runs on the full suite only (steps.scope.outputs.reduced == 'false'), with the test step's shell, so the same user and home. The Run tests steps are not edited.
  • It is best effort: the script exits 0 whatever happens, has a 240 s budget, and the step carries continue-on-error and a 5-minute timeout. A file it did not cache is rewritten by pytest.
Design point Choice
Files prewarmed The shard's own test files: under testpaths, matching python_files, owner decided by scripts.ci_file_shards.file_shard with the same pin file. Plus every conftest.py under the rootdir and testpaths
Not prewarmed Modules pytest rewrites only because something imports or registers them: another shard's test module imported by one of this shard's tests, an installed module whose name matches python_files (aiohttp/test_utils.py), pytest_plugins, register_assert_rewrite. Finding them needs an import. pytest rewrites them as before (70 files on shard 1)
Conftest ignores Not consulted (that needs the conftest import). A file pytest later ignores gets a pyc nobody reads. Discovery stays pytest's own
pyc format, name, directory pytest's own functions: _rewrite_test, _write_pyc, PYC_TAIL (<name>.cpython-312-pytest-9.0.3.pyc), get_cache_dir
Cache root pycache_prefix() mirrors _redirect_bytecode_cache in test/conftest_floor_configure.py: $XDG_CACHE_HOME or <account home>/.cache, then kirocrew/pycache. Each process sets sys.pycache_prefix to it, so get_cache_dir builds the path, drive letter handling included. A test pins the two functions equal
Rootdir spelling os.getcwd(), as pytest uses, since the cache directory is derived from the absolute source path

Local evidence only (dev host, 32 cores, loaded; shard 1 of 8, 992 files, 33,590 tests; pytest --collect-only -q -n0 --no-cov, fresh cache root per arm):

Arm Round 1 Round 2
Cold collection 120.5 s 122.4 s
Prewarm, 8 processes 3.8 s 4.4 s
Collection after prewarm 53.1 s 60.7 s
Collection, fully warm cache 36.8 s 40.6 s

With 32 processes the prewarm took 2.0 s. In a traced run after a prewarm (--debug), pytest reported "found cached rewritten pyc" for 985 modules and "rewriting" for 70, none of them prewarmed files. A 2-worker xdist collection after a prewarm left all 992 prewarmed pycs untouched and added the same 70, so workers read the cache too. These numbers are one process without coverage on a developer host. They do not predict the CI result.

Backwards compatibility

Compatible: one added CI step that only writes bytecode under the runner's cache directory; no product code moves.
Removes nothing: the test steps, their scopes and the files each shard collects are unchanged.

Tests

test/test_ci_rewrite_prewarm.py (new, 11 tests):

  • Prewarm a temp tree, run pytest on it with --debug: the rewriter traces "found cached rewritten pyc" for every test file and "rewriting" for none, and every pyc, conftest ones included, is byte- and mtime-identical afterwards.
  • Negative controls: without the prewarm, and with the prewarm written under another cache root, the same trace shows "rewriting" for every file and no cache hit.
  • No import: a tree whose test module and conftests write a marker at import time shows no marker after the prewarm, sys.modules holds no kiro_crew, conftest or test module, and the markers do appear when pytest runs.
  • A file with a syntax error is reported, skipped and the exit status is 0. Bad arguments and a missing pin file exit 0.
  • The four shards' file lists are disjoint, cover every test file, follow file_shard and a pin.
  • Discovery follows python_files, testpaths and pytest's default norecursedirs.
  • The cache root equals the one _redirect_bytecode_cache sets.
  • Workflow: one prewarm step in each job, directly before Run tests, full suite only, same shell, same shard flags and pin file, continue-on-error, no actions/cache step, and no other job runs it.

Existing pins re-run green: test/test_ci_file_shards.py, test/test_ci_pytest_progress.py, test/test_ci_windows_canary.py.

Manual verification

Ran the script on this checkout's shard 1 with a fresh cache root, then collected the shard (table above). Not run on Windows locally: the Windows path shape and spawn are exercised by this PR's Windows shards, which run the new test file and the new step.

Real-machine verification

N/A -- CI workflow only; measured on CI runners.

Related Issues

Closes #39368

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

AI-assisted, opened by Claude Code

Every xdist worker of a backend shard collects the whole shard and, on
a fresh runner, rewrites the asserts of each test file because no
cached pyc exists. A new step before the full-suite test step writes
those pycs once, one process per core, with pytest's own rewrite
functions, under the cache root the rootdir conftest sets. It parses
and compiles source and imports no test module, conftest or kiro_crew,
unlike the serial --collect-only prewarm that measured slower.

The step is best effort: it exits 0 whatever happens and pytest
rewrites any file it did not cache. It does not choose what a shard
collects. This is an experiment with a ship gate, measured on the
pull request's own full-suite run (#39368).
@Zedmor Zedmor added the ci-full-run Force the full backend test matrix instead of the reduced scope; read on the next push label Nov 19, 2026
@Zedmor Zedmor self-assigned this Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ⚠️ review incomplete

GPT 6.1 did not produce a complete verdict for a1c4ca9901b5ad317ff8cbfe1af0cfcd8f608e96; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 6.1 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a1c4ca9901b5ad317ff8cbfe1af0cfcd8f608e96 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision a1c4ca9901b5ad317ff8cbfe1af0cfcd8f608e96 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a1c4ca9901b5ad317ff8cbfe1af0cfcd8f608e96 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Heads-up: this PR edits .github/workflows/ci.yml, and #39327 also edits that file and is still open. Please rebase this branch after #39327 merges, resolve any ci.yml overlap, and let the checks run again on the rebased head before merge.

AI-assisted, sent by Kiro Crew Captain

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full-run Force the full backend test matrix instead of the reduced scope; read on the next push readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: try a parallel, rewrite-only assertion-rewrite prewarm before xdist starts (ship only if measured faster)

2 participants